1. Technical Field
The invention relates to testing electronic circuitry. More particularly, the invention relates to a method and apparatus for verification of such electronic circuitry in integrated circuits, boards, and systems.
2. Description of the Prior Art
Design verification is the process of determining whether a particular hardware and software combination, such as an integrated circuit (IC), exactly implements the requirements defined by the IC""s specification. As IC designs become more complex, the likelihood of design errors increases. However, design verification also becomes increasingly difficult and time consuming.
A typical prior art design verification scheme is devised in accordance with the device""s specifications. A comprehensive test plan for the design is then devised and implemented. The verification scheme and test plan are usually both device-specific, and must therefore be revised or generated anew for each device to be tested.
Verification of a device under testing (DUT) frequently requires the DUT to be tested through integration cycles, as well as through specification changes. Additionally, the tester must determine the coverage of a test, for example, between subsystem and full-system verification, to determine if the testing is to be continued or has already been completed.
Verification testing may be performed on the actual device, or on a simulation model of the device. For purposes of explanation only, the following discussion will be limited to simulation models.
As systems become larger and more complicated, functional verification consumes a larger percentage of engineering resources. Furthermore, the prior art verification tools do not achieve the same level of abstraction as the design specification used for verification. Thus, the test generation tools may not allow the use of the abstract data types required to model a design.
Hardware Description Languages (HDLs) such as Verilog and VHDL are used to verify a design through the comparison of at least two design representations. Low-level netlists also perform a similar type of verification procedure. In this procedure, HDL and Netlist views of a design are simulated. Input stimuli to the simulator can come from a variety of sources. Checking is done by comparing results or behavior with the expected results that are either concluded by the designer or computed by a predictor simulator.
HDLs are designed to describe hardware at higher levels of abstraction than gates or transistors. However, such HDLs are typically not designed for verification. Thus, a verification engineer must often design a verification testbench that includes interfaced HDL programming, C or C++ programming, and various scripts.
In a system-level simulation approach, a designer builds a hierarchical, modular model of the entire device system. Test vectors are then developed to verify the functionality of the system. A test vector is a string of binary digits that is applied to an input of a circuit. The test vector is used to compare the circuit""s output to the expected results.
In the prior art, the solution to increasing verification productivity has been to introduce increasingly faster simulators. Modeling and simulation can be accelerated through use of, for example, hardware modelers, commercial bus-functional models, simulator accelerators, and Field Programmable Gate Array (FPGA)-based Application Specific Integrated Circuit (ASIC) emulators.
However, the prior art mechanisms have not adequately addressed the reduction of verification development time. Verification engineers are still needed to write functional tests, develop diagnostic suites, determine test coverage, and analyze the results.
Verification engineers typically use combinations of deterministic, static, and dynamic tests to construct testing tools. In deterministic testing, each test is developed, checked, and executed manually. While some level of automation can be introduced into the testing process, deterministic testing still generally requires significant efforts of skilled personnel.
Deterministic tests are usually design-specific. Therefore, re-use of such a verification procedure is difficult, if not impossible. With the time, cost, and personnel constraints imposed by current marketing conditions, functional coverage by means of deterministic testing alone is usually low.
A static testbench is a program that drives pre-computed test vectors into the DUT simulator, and/or checks outputs after the simulation is completed. Such static testbench is frequently used to increase functional coverage during design development. A static testbench can include elements such as HDL Behavioral Functional Models (BFMs), monitors, and memories, as well as random test generators and post-run-checkers.
Generally, a static testbench is used to check the results of a test after the test has been completed. Thus, an error in the test is not detected until after the test is finished. As a result, the internal state of the device at the point of the error is not determined. To analyze signals and register states, the test must be re-run to the point at which the error occurred. This procedure consumes simulation cycles, and can require the expenditure of considerable time, especially during long tests.
A static testbench usually requires as least as much, and frequently twice the amount of code as the HDL model to which it is applied. The static testbench code is difficult to manage and control. Additionally, a static testbench is almost completely design-specific, and contains fixed code, written for one design. Thus, a static testbench is rarely re-usable for a next-generation design.
Parameter-driven test generation can be used to overcome the limitations of a static testbench. In this procedure, a program is written, for example in the C language, to generate test vectors. These test vectors are stored to a memory storage device, such as a hard drive, prior to the commencement of the test. C code is versatile and allows the creation of functions that can be reused for different device designs.
Parameter-driven test generation allows directed random generation. For example, a directed random generator can be used to test a communications device by creating large numbers of test packets meeting known parameters.
Parameter-driven test generation can support a low level of interaction with the HDL model. This is because the test program can be directed to send new stimuli to the simulator only after the occurrence of particular events. However, parameter-driven test generation is static. Thus, parameter-driven test generation cannot shape such stimuli in response to a model""s state. It is therefore often difficult to generate exactly the correct stimulus to bring about the required state of the device for design errors detection.
For example, in certain memory controllers, a write to the master Direct Memory Access (DMA) count register should abort the current transaction. However, using a static procedure, it is extremely difficult to generate a test to verify that this will happen even if the write occurs in the last clock cycle of the current transaction.
A program that generates test vectors in concurrence with the DUT simulation and while being controlled by the state feedback of the simulated device is known as a dynamic testbench. A dynamic testbench can therefore be used to shape test vectors in response to device states. In such dynamic testbench, the test programs interact with the HDL model through an Application Program Interface (API) supported by the underlying simulator.
However, the implementation of such API is not a simple task. API implementation can differ among simulators. For example, each VHDL simulator has a different API. While Verilog uses a standard Procedural Language Interface (PLI), the interface is not always the same, especially between event-driven and cycle-based simulators. Additionally, if more than one simulator is used, such as a cycle-based simulator at the RTL level and an event-driven simulator at the gate level, several such interfaces must be developed.
The API permits direct function calls to be made from a dynamic testbench to the HDL model. These function calls can drive or sample the device model""s input and output (I/O). After the occurrence of any event, the dynamic testbench code can use information sampled from the device model to shape the next set of stimuli applied to the model.
A dynamic testbench is used to advantage in verifying the functionality of events that occur at indeterminate times. Thus, a dynamic testbench can monitor a DMA operation and write a new value to the DMA count register in its last clock cycle, to verify that the current transaction is aborted.
There are several known disadvantages associated with the prior art dynamic testbenches. Such testbenches can be expensive to develop and maintain. Development time for a dynamic testbench can occasionally approach several months. Furthermore, dynamic testbenches are generally architecture-specific and parameter-driven.
In the prior art, procedures such as toggle testing and code coverage have been used to determine functional coverage. Toggle testing indicates the number of gates and flip-flops that are touched by a test. However, toggle testing does not indicate whether a test covers all system states.
Code coverage indicates how many lines of code in an HDL model are executed. Many different functions in a design can activate the same flip-flop or line of code. However, code coverage does not indicate whether all such conditions have been tested. Additionally, toggle testing and code coverage cannot detect any omitted functionalities. Therefore, it is difficult, if not impossible, to provide complete functional coverage using these prior art procedures.
Device functions must often be tested with regard to a hardware apparatus, such as the design of a chip or board that is under the control of a central processing unit (CPU). In such case, the CPU is not a part of the DUT, but the CPU can run certain programs that invoke the DUT. Examples of such scenario include:
a driver program in C programming language controlling an ATM switch;
the software of an answering machine controlling the answering machine chip(s); and
a C diagnostic program checking an MPEG compression board.
The DUT design is verified by running the software against the hardware. However, it can be desirable to perform such verification tests while the hardware is still in simulation phase. One method for verifying the DUT with regard to a hardware apparatus is to use a cycle-accurate model of the CPU on which the software is to run. This model can be a hardware model, or can be simulated in Verilog, or C.
The code for the CPU model is compiled and loaded into, for example, a Verilog memory. The model then executes the compiled code while simultaneously running the DUT. However, this process is time consuming, and is subject to compiling and configuration errors.
Formal verification is a process for statically checking a model to determine whether the model is an accurate representation of a DUT. Thus, no simulation is run in a formal verification procedure. A formal verification tool usually first reads a gate-level or register transfer level (RTL) model and subsequently reads assertions in the verification tool""s language. The formal verification tool then verifies whether the given model obeys the assertions. However, because a verification tool""s language is typically not standardized, and can be difficult to use, formal verification is frequently a complex process.
Additionally, formal verification techniques are subject to a state explosion problem. This problem can occur when the necessary representation of the DUT logic consumes too much memory, thereby preventing the formal verification process.
It would therefore be an advantage to provide an automatic an dynamic verification method and apparatus that is reusable for all applications. It would be a further advantage if such method and apparatus reduced test development time. It would be yet another advantage if such method and apparatus facilitated the creation and verification of hardware models.
The invention provides a method and apparatus for functionally verifying an integrated circuit design. The invention uses a verification-specific object-oriented programming language to verify a device under testing (DUT). Such devices can include modules, chips, simulations, and systems, including systems having embedded software. A testbench created using the invention can be reused to test a device during each stage of design development.
The invention is adapted for integration with Verilog, VHDL, and C functions. A modular system environment ensures interaction with any supported simulator through a unified system interface that supports multiple external types. The unified system interface can also be adapted for custom modeling environments.
A hardware-oriented verification-specific object-oriented programming language is used to construct and customize the verification tests. The structure and components of this language are shaped to provide those elements needed to stimulate and observe a model of a hardware device. The verification-specific object-oriented programming language is extensible, allowing for incremental development and patching.
The invention includes a test generator module for automatically creating a device verification test from a functional description. The test generator can be constrained to generate tests for specific subsets of the design""s functionality. Thus, some tests can focus on a specific feature in the design, while other tests can address broader functional scopes. The invention can also be provided with extensions to perform optional testing, such as co-verification or formal verification of a simulation model.
A test suite designed by the invention can include any combination of statically and dynamically-generated tests. The user has full control of the random generation process and can vary the test parameters at any point throughout the test generation.
The invention supports random stability. Thus, a change in a local constraint generally has a local effect on test generation, while the rest of the test is unaffected. The random number generator is both platform and simulator-independent.
Data checks are performed to verify the relation among different data elements, or to verify data against a high level reference model. The invention can perform any combination of static and dynamic checks. When using both dynamic generation and dynamic checking, the test generator module and the checker can constantly synchronize. Thus, temporal checks can be precisely targeted.
Incremental testing permits the gradual development of test suites throughout the design development process. The invention can be used in architectural modeling and for module verification. Additionally, known bugs can be intentionally bypassed to permit verification of other device functions.
The invention provides reports of functional coverage statistics at the architectural, module, or system levels. Cross coverage reports can also be generated for later analysis.
The invention includes a graphical user interface (GUI) that facilitates the verification process. The GUI permits the user to direct and control the invention, as well as to view the status of a testing procedure.
A source level debugger uses a debugging GUI that enables debugging with a data input device, such as a mouse or pointer device. Hyperlinks to related elements can be provided. Data structures, lists of events, and their interrelations can be displayed for interactive debugging by the user. The user can set breakpoints in the hardware-oriented verification-specific object-oriented programming language code, and can observe and change variable values inside both the verification-specific object-oriented language and the HDL code. The user can also use the invention to generate customized reports detailing the debugging process.
The invention supports High-Level Verification Automation (HLVA) facilities, such as the ability to split and layer architecture and test files. Thus, the invention can reuse both verification environments and test suites. Such recycling can reduce the time expended in verification and therefore increase productivity.